Smart article visual communication based on facial movement

ABSTRACT

A smart mask includes a first material layer, at least one display, a first sensor, and a control module. The first material layer is configured to cover a portion of a face of a person. The at least one display is connected to the first material layer and configured to display images over a mouth of the person. The first sensor is configured to detect movement of the mouth of the person and generate a signal indicative of the movement of the mouth. The control module is configured to receive the signal and display the images on the display based on the movement of the mouth.

FIELD

The present disclosure relates to clothing technologies with embeddedelectronics and more particularly to electronics-augmented face masks.

BACKGROUND

During a pandemic, people are often isolated to prevent the spread of,for example, viral and/or bacterial infections. People may wear facemasks to reduce risk of infection themselves and/or to reduce risk ofinfecting others. Wearing of face masks can become widespread during apandemic, such that many institutions (e.g., retail shops, stores,services, etc.) will not allow people to enter the institutions unlessthey are wearing a face mask, whether as a corporate decision or as alegal requirement.

The background description provided here is for the purpose of generallypresenting the context of the disclosure. Work of the presently namedinventors, to the extent it is described in this background section, aswell as aspects of the description that may not otherwise qualify asprior art at the time of filing, are neither expressly nor impliedlyadmitted as prior art against the present disclosure.

All external document references are incorporated by reference in theirentirety.

SUMMARY

A smart mask includes a first material layer, at least one display, afirst sensor and a control module. The first material layer isconfigured to cover a portion of a face of a person. The at least onedisplay is connected to the first material layer and configured todisplay images over a mouth of the person. The first sensor isconfigured to detect movement of the mouth of the person and generate asignal indicative of the movement of the mouth. The control module isconfigured to receive the signal and display the images on the displaybased on the movement of the mouth.

In other features, the smart mask further includes a second materiallayer, a third material layer, and a power source. The power source isconnected to the second material layer and powering the control module.The first sensor is connected to the third material layer. The signal isindicative of movement of a first portion of the mouth of the person.The control module is configured to display the images on the displaybased on the signal.

In other features, the first sensor includes an adhesive layer forattaching to the face of the person. In other features, the smart maskfurther includes a second sensor configured to generate a signalindicative of movement of a second portion of the mouth. The controlmodule is configured to display the images on the display based on thesignal indicative of the movement of the second portion of the mouth.

In other features, the at least one display is embedded in the firstmaterial layer or overlaid on the first material layer. In otherfeatures, the at least one display is disposed in one or more pockets onthe first material layer.

In other features, the at least one display includes multiple displays.Each of the displays is configured to display one or more images basedon the detected movement of the mouth. In other features, the at leastone display is perforated.

In other features, the smart mask further includes a second materiallayer and spacers disposed to separate the at least one display from thesecond material layer. In other features, the smart mask furtherincludes multiple layers including the first material layer. The layersinclude channels for air flow.

In other features, the smart mask further includes a filter includingmultiple sensors including the first sensor. The sensors are configuredto generate signals indicative of facial movements of the personincluding the movement of the mouth or of utterances. The control moduleis configured to display the images based on the signals received fromthe sensors.

In other features, the first sensor is a graphene sensor. In otherfeatures, the smart mask further includes multi-purpose sensors, whereeach of the multi-purpose sensors include antimicrobial material capableof generating a sensor signal (e.g., resistance, capacitance, current,voltage, etc.). In other features, at least one of the multi-purposesensors includes graphene-based nanomaterials having antibacterialfeatures. In other features, the multi-purpose sensors include at leastone of a nanotube sensor or a graphene health sensor.

In other features, the at least one display includes light emittingdiodes. The light emitting diodes are arranged to track movements of themouth. The images are low resolution images generated by the lightemitting diodes.

In other features, the smart mask further includes an audio sensorconfigured to detect sounds or utterances generated by the mouth of theperson. The control module is configured to display the images based onthe detected sounds.

In other features, the control module is configured to perform anartificial training process or method with one or more devices such thatat least one of the control module or the one or more devices are ableto map detected facial movements of the person to the images for displayon the at least one display.

In other features, a system includes a first article and a secondarticle. The first article includes a first material layer, at least onesensor and a control module. The at least one sensor is connected to thefirst material layer and configured to generate a sensor signalindicative of at least one movement of a mouth of a person or soundgenerated via the mouth of the person. The control module is configuredto transmit the sensor signal from the first article indicating the atleast one movement of the mouth of the person or sound generated via themouth of the person. The second article is separate from the firstarticle and includes a second material layer, a transceiver and at leastone display. The second material layer configured to cover a portion ofa body of the person. The transceiver is configured to receive thesensor signal. The at least one display is connected to the secondmaterial layer and configured to display images on the at least onedisplay based on the at least one movement of the mouth of the person orsound generated via the mouth of the person.

In other features, the first article is a mask. In other features, thesecond article is a mask. In other features, the images include at leastone of images of the mouth or other images (e.g., cartoon characters,emoticons, text, sign language graphics, etc.).

In other features, a smart article is provided and includes a firstmaterial layer, at least one display and a control module. The firstmaterial layer is configured to cover a portion of a body of a person.The at least one display is connected to the first material layer andconfigured to display images. The control module is configured to detectmovement of a mouth of the person and display the images on the displaybased on or in response to the movement of the mouth.

In other features, the smart article further includes a second materiallayer, a third material layer, and a power source. The power source isconnected to the second material layer and powering the control module.The first sensor is connected to the third material layer and configuredto generate a signal indicative of movement of a first portion of themouth of the person. The control module is configured to display theimages on the display based on the signal.

In other features, the first sensor comprises an adhesive layer forattaching to a face of the person. In other features, the smart articlefurther includes a second sensor configured to generate a signalindicative of movement of a second portion of the mouth. The controlmodule is configured to display the images on the display based on thesignal indicative of the movement of the second portion of the mouth.

In other features, the at least one display is embedded in the firstmaterial layer or overlaid on the first material layer. In otherfeatures, the at least one display is disposed in one or more pockets onthe first material layer.

In other features, the at least one display includes multiple displays.Each of the displays is configured to display one or more images basedon the detected movement of the mouth. In other features, the at leastone display is perforated or porous.

In other features, the smart article further includes a second materiallayer and spacers. The spacers are disposed to separate the at least onedisplay from the second material layer. In other features, the smartarticle further includes layers including the first material layer. Thelayers include channels for air flow.

In other features, the smart article further includes a filter includingsensors. The sensors are configured to generate signals indicative offacial movements of the person including the movement of the mouth. Thecontrol module is configured to display the images based on the signalsreceived from the plurality of sensors.

In other features, the smart article further includes at least onegraphene sensor configured to generate a signal indicative of themovement of the mouth. The control module is configured to display theimages on the display based on the signal.

In other features, the at least one display includes light emittingdiodes. The light emitting diodes are arranged to track movements of themouth. The images are low resolution images generated by the lightemitting diodes.

In other features, the at least one display includes light emittingdiodes configured to track movements of the mouth.

In other features, the smart article further includes an audio sensorconfigured to detect sounds generated by the mouth of the person. Thecontrol module is configured to display the images based on the detectedsounds.

In other features, the control module is configured to perform anartificial training process with one or more devices such that at leastone of the control module or the one or more devices are able to mapdetected facial movements of the person to the images for display on theat least one display.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description, the claims, and the drawings.The detailed description and specific examples are intended for purposesof illustration only and are not intended to limit the scope of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description and the accompanying drawings.

FIG. 1 is a functional block diagram of an example smart mask ecosystemin accordance with the present disclosure.

FIG. 2 is a functional block diagram of an example smart mask having asmart mask visual communication system in accordance with the presentdisclosure.

FIG. 3 is a functional block diagram of an example mobile deviceimplementing a smart mask application in accordance with the presentdisclosure.

FIG. 4 is a functional block diagram of an example smart wearablearticle in accordance with the present disclosure.

FIG. 5A is a back view of an example smart mask including a singlestretch sensor in accordance with the present disclosure.

FIG. 5B is a front view of the smart mask of FIG. 5A including lightemitting diodes (LEDs) in accordance with the present disclosure.

FIG. 5C is a front view of the smart mask of FIG. 5A illustratingcertain ones of the LEDs in an ON state due to a person's mouth being ina closed state.

FIG. 5D is a front view of the smart mask of FIG. 5A illustratingcertain ones of the LEDs in an ON state due to a person's mouth being inan open state.

FIG. 6 is a front view of another example smart mask including multipleflexible displays in accordance with the present disclosure.

FIG. 7 is a front view of another example smart mask including a singleflexible display in a transparent pocket in accordance with the presentdisclosure.

FIG. 8 is front view of another example smart mask including multipleflexible displays in transparent pockets in accordance with the presentdisclosure.

FIG. 9 is a side cross-sectional view of a portion of an example smartmask including an embedded display and sensors with skin adheringcontacts in accordance with the present disclosure.

FIG. 10 is a side cross-sectional view of a portion of an example smartmask including a single stretch sensor and an overlay layer including adisplay in accordance with the present disclosure.

FIG. 11 is a side cross-sectional view of a portion of an example smartmask including multiple sensors and an overlay layer including a displayin accordance with the present disclosure.

FIG. 12 is a side cross-sectional view of a portion of an example smartmask including an overlay layer, a sensor and a perforated display inaccordance with the present disclosure.

FIG. 13 is a side cross-sectional view of a portion of an example smartmask including multiple sensors and an overlay layer with a displayspaced away from other layers in accordance with the present disclosure.

FIG. 14 is a side cross-sectional view of a portion of an example smartmask including multiple sensors and a cooling layer with channels forcooling a display in accordance with the present disclosure.

FIG. 15 is a back view of an example face mask including a filter withembedded sensors in accordance with the present disclosure.

FIG. 16A is a front view of a mouth of a person in a silent and neutralposition.

FIG. 16B is a front view of the mouth of FIG. 16A with overlaidlandmarks and an origin in accordance with the present disclosure.

FIG. 16C is a front view of the mouth of FIG. 16A with the overlaidlandmarks of FIG. 16B moved to different locations associated with mouthmovement and relative to the origin in accordance with the presentdisclosure.

FIG. 16D is an example diagram illustrating positioning of landmarksrelative to the origin in accordance with the present disclosure.

FIG. 17 illustrates an example article operation method according to thepresent disclosure.

FIG. 18 illustrates an example mobile device operation method accordingto the present disclosure.

FIG. 19 illustrates an example central control station operation methodaccording to the present disclosure.

In the drawings, reference numbers may be reused to identify similarand/or identical elements.

DETAILED DESCRIPTION

While face masks aid in protecting people from air-borne diseases, facemasks often interfere with person-to-person communication. For example,a face mask can hide significant portions of a face of a personspeaking, which can interfere with the ability of others to interpretthe facial expressions of the person speaking. This limited ability tosee and interpret facial expressions also limits the full breadth ofcommunication.

The examples set forth herein include smart masks and other smartarticles. As used herein, the term “article” refers to any wearableelement, such as pants, hats, shirts, gloves, shoes, etc. or portionsthereof. A smart article may include one or more sensors for detectingbody movements, such as and including mouth and/or cheek movements, andone or more displays for displaying images and/or video of acorresponding person's face showing movements of the person's face asthe movements occur (i.e. in real-time). Each smart article may alsoinclude one or more other sensors, such as audio sensors for detectingverbal sounds created by the person, strain sensors, compression sensor,piezoelectric sensors, thermal sensors, or other types of sensor. Thesmart article or other device in communication with the smart articlemay map data received from the sensors to body movements, facialmovements for example and display the facial movements on an electronicdisplay.

As an example, a face mask (hereinafter referred to as a “mask”) or maskoverlay may display a digital representation of a person's face, wherethe digital representation includes a rendering that mimics or matchesthe person's face movements as the person speaks and/or emotes. Thisaids a person listening to the speaker in interpreting the speaker. Ahigh-level example for rendering an image on a mask includes: creating adigital and flexible display overlay that fits over a mask (or as partof a mask); attaching the display to the mask; connecting an articlecontrol module to the display; and showing on the display in at leastnear real-time what the person's face and/lips are doing includingmovements of lips, tongue, cheeks, nose, etc. The display may withsufficient resolution render what the person's actual face looks likewhile the person speaks, utter sounds, and/or emotes.

FIG. 1 shows an example smart mask ecosystem 100 that may include acentral control station 102, a smart mask 104, a mobile device 106, andone or more other smart wearable articles 108. The central controlstation 102 may communicate with any of the smart mask 104, the mobiledevice 106 and the smart wearable articles 108 as further describedbelow in association with the examples of FIGS. 2-19. The centralcontrol station 102 may include a central control module 110, atransceiver 112, a display 114, sensors 116, a memory 18 and a userinterface 120. The central control module 110 may collect sensor data,map the data to facial movements and provide results of the mapping toand of the devices 104, 106, 108. The central control module 110communicates with the devices 104, 106, 108 via the transceiver 112. Thememory 118 may store software for mapping the sensor data disclosedherein. Example mapping modules and applications are shown in FIGS. 2-4with respect to the devices 104, 106, 108 and described below and may besimilarly implemented in the central control station 102. The mappingsoftware is executed by the central control module 110.

The sensors 116 may include various types of sensors capable ofacquiring one or more modalities of sensor data. Example sensors caninclude one or more cameras, audio sensors (e.g., microphones), thermalsensors, accelerometers, and/or other sensors for detecting and/orrecording facial movements. Audio and video may be initially recordedfor a person and stored as a reference (or reference data) and used forfuture mapping as discussed further below. Additional real-time audiomay also be detected during subsequent use and used for mapping facialmovements while a person is, for example, speaking. The reference data,the real-time audio and/or other data collected by the devices 104, 106,108 may be translated to facial movements and provided to the devices104, 106, 108 for display an electronic display on one or more of thedevices 104, 106, 108. Some or all of the mapping may occur at any ofthe devices 104, 106, 108 as described below. Examples of the devices104, 106, 108 are shown and described with respect to FIGS. 2-19.

The display 114 may be a touchscreen or other type of display. Thesensors 116 may include audio sensors, cameras, accelerometers, etc. Thesensors 116 may be used for recording and training purposes and/or formapping purposes as further described below.

FIG. 2 shows an example smart mask 200 that may represent the smart mask104 of FIG. 1 and includes a smart mask visual communication system 202.The smart mask visual communication system 202 may include a maskcontrol module 204, one or more displays (one display 206 is shown inFIG. 2), a memory 208, a transceiver 210 and one or more sensors 212.The mask control module 204 may include an interface 213, a sensormodule 214, an audio recognition module 216 and a mapping module 218.The interface 213 may include channels connected to the sensors 212. Theinterface 213 may include one or more channels connected respectively toone or more sensors, where X refers to the number of channels andrepresents any practical number of channels (e.g., at least 1 channel,at least 2 channels, at least 10 channels, and so on) as required by thedesired implementation.

The sensor module 214 may collect sensor data from the sensors 212and/or from any other devices in the ecosystem 100 of FIG. 1. The audiorecognition module 216 may execute audio recognition software totranslate detected audio into text, which may then be used by themapping module 218 for mapping sensor data to image data for display onthe display 206, one or more other displays of the smart mask and/orother device (e.g., smart articles) of the ecosystem 100. The mappingmodule 218 may map sensor data to image data for display on one or moredisplays. This may including mapping detected movements to display data,which may be based on the output of the audio recognition module 216.The transceiver 210 may be used to communicate with any of the devicesin the ecosystem 100. One should appreciate the selection of images todisplay can depend on one or more sensor data modalities (e.g., audiodata, stress sensor data, strain sensor data, thermal data, pressuresensors, etc.).

As an example, several small sensors may be included that are able tosense movement of the mask while a person is talking or emoting. Thesensors 212 may include piezoelectric sensors, thermal sensor, stretchsensors, etc. As another example, a predetermined number of sensors(e.g., four sensors) may be included; one attached to each of thestrings of the mask. The sensors 212 may include sensors that attach toa person's face. The sensors 212 may be disposed in material (or layers)of the mask 200. In one embodiment, sensors are included in or operateas one or more filters of the mask 200.

FIG. 3 shows an example mobile device 300 may represent the mobiledevice 106 of FIG. 1 and may include a mobile device control module 302,a display 304, a memory 306, a power source 307, a transceiver 308 andsensors 309. The mobile device 300 may be a cell phone, a laptopcomputer, a tablet, a wearable device (e.g., a smart watch), a dedicatedmask controller, etc. The mobile device control module 302 executes asmart article application 310 for mapping collected data to facialmovement images. The mobile device control module 302 may include amapping module 311, a sensor module 312 and/or an audio recognitionmodule 313. The mapping module 311 may be referred to as a classifierand include a face movement-to-text mapping module 314, a sensoroutput-to-text mapping module 316, a text-to-facial display mappingmodule 318, and other mapping modules 320, which may be implemented aspart of the smart article application 310. Thus, mapping module 316 isconsidered to convert one or more sensor modalities to one or more otherdisplay modalities (e.g., 2D digital rendering of a face, 3D digitalrendering of a face, displayed text, displayed sign language, displayedemoticons, displayed emotes, etc.).

The modules 311, 312, 313, 314, 316, 318, 320 are provided as examplesonly, the mobile device 300 may include other modules. The facemovement-to-text mapping module 314 may map detected face movements,which may be based on data received from other devices (e.g., one ormore of the devices 104, 108 of FIG. 1), to text. The sensoroutput-to-text mapping module 316 may map data from sensors 309 and/orsensors of other devices (e.g., one or more of the devices 104, 108 ofFIG. 1) to text. The text-to-facial display mapping module 318 may mapthe text from the modules 314, 316 to facial images to display. One ofthe other mapping modules 320 may map received and collected datadirectly to facial images. The described mappings may be based onreference data 321 and the collected sensor data (designated 322) storedin the memory 306. Based on this example, one should appreciate themapping can include a direct mapping of the sensor data to image data(i.e., audio-to-image) or an indirect mapping to image data via anotherdata modalities (i.e., audio-to-text and then text-to-image).

The sensor module 312 may collect sensor data from the sensors 309and/or from any other device in the ecosystem 100 of FIG. 1. The audiorecognition module 313 may execute audio recognition software totranslate detected audio into text, which may then be used by themapping module 311 for mapping sensor data to image data for display onthe display 304, one or more other displays of a smart mask, one or moredisplays of smart articles, and/or other device of the ecosystem 100.Example techniques that can be adapted for use with audio recognition orlanguage recognition includes those disclosed in one or more of U.S.Pat. Nos. 10,347,240; 8,229,728; 8,374,871; 8,478,578; 8,583,416; and9,026,441. The mapping module 311 may map sensor data to image data fordisplay on one or more displays. This may include mapping detectedmovements to display data, which may be based on the output of the audiorecognition module 313. The transceiver 308 may be used to communicatewith any of the devices in the ecosystem 100.

The display 304 may be a touchscreen and/or other type of display. Thesensors 309 may include audio sensors, cameras, accelerometers, locationsensors, etc. The sensors 309 may be used for recording and trainingpurposes and/or for mapping purposes as further described below.

FIG. 4 shows an example smart wearable article 400, which may representany of the smart mask 104 and smart articles 108 of FIG. 1. The smartwearable article 400 may include a smart mask visual communicationsystem 402, which may include an article control module 404, one or moredisplays (one display 406 is shown in FIG. 4), a memory 408, atransceiver 410, one or more sensors 412 and a power source 414. Thearticle control module 404 may be suitable for wearable implementationsand be based on wearable technologies. The article control module 404may have a computing platform specifically suited for being sown intoclothing or fabric.

A smart mask device may include a smart mask controller operating basedon a computing platform highly suitable for wearable technologies. Acouple of examples include the Raspberry PI computing platform (see URLwww.raspberrypi.org) and the Arduino computing platform (see URLwww.arduino.cc). Still further, a computing platform that isspecifically suited for being sown into clothing or fabric includes thewearable controller offered by Adafruit and based on the Arduinoplatform (see URL www.adafruit.com/category/65).

The article control module 404 may include an interface 413, a mappingmodule 415, a sensor module 416, and an audio recognition module 418.The mapping module 415 may be referred to as a classifier and include aface movement-to-text mapping module 420, a sensor output-to-textmapping module 422, a movement-to-image mapping module, a text-to-facialdisplay mapping module 424 and/or other mapping modules 426.

The interface 413 may include channels connected to the sensors 412. Theinterface 413 may include one or more channels connected respectively toone or more sensors, where Y refers to the number of channels.

The modules 404, 415, 416, 418, 420, 422, 424, 426 are provided asexamples only, the smart wearable article 400 may include other modules.The face movement-to-text mapping module 420 may map detected facemovements, which may be based on data received from other devices (e.g.,one or more of the devices 104, 108 of FIG. 1), to text. The sensoroutput-to-text mapping module 422 may map data from sensors 412 (e.g.,one or more of the devices 104, 108 of FIG. 1) to text. Thetext-to-facial display mapping module 424 may map the text from themodules 420, 422 to facial images to display. One of the other mappingmodules 426 may map received and collected data directly to facialimages. The described mappings may be based on reference data 421 andthe collected sensor data (designated 423) stored in the memory 408.Although some embodiments leverage a movement to text mapping module, itshould be appreciated that other embodiments map movements to images.

The sensor module 416 may collect sensor data from the sensors 412and/or from any other devices in the ecosystem 100 of FIG. 1. The audiorecognition module 418 may execute audio recognition software totranslate detected audio into text, which may then be used by themapping module 415 for mapping sensor data to image data for display onthe display 406 one or more other displays of the smart mask and/orother device of the ecosystem 100. The mapping module 415 may map sensordata directly or indirectly to image data for display on one or moredisplays. This may including mapping detected movements to display data,which may be based on the output of the audio recognition module 418.The transceiver 410 may be used to communicate with any of the devicesin the ecosystem 100.

Referring again to FIGS. 1-4, the smart masks disclosed herein mayrender an image output of a portion of a person's face thereby allowingothers to see, for example, the person's mouth, cheeks and/or nose moveor possibly see the person's emotes. Each smart mask and smart article(or article) may be referred to as a computing device having a controlmodule (e.g., a control circuit and/or processor, FPGA, ASIC, GPU, etc.)and memory. The control module executes software instructions stored ina non-transitory computer readable memory (e.g., RAM, ROM, flash, SSD,HDD, etc.) to perform operations disclosed herein. The articles mayinclude an electronic display that generates an output based on sensordata received via sensors. In addition to including interfaces forsensors, the control modules and/or articles may include interfaces forother devices, such as transceivers. The transceivers may be used fordata uploads, downloads, and/or other communication signals. Thecommunication interfaces may include wired interfaces (e.g., universalserial bus (USB), recommended standard (RS)-232, general purpose inputoutput (GPIO), etc.) or wireless interfaces (e.g., 802.11, wirelessgigabit (WiGIG), BlueTooth®, Zigbee®, near-field communication (NFC),etc.).

In some embodiments, multiple articles may cooperate together, possiblyvia wireless computing channels (e.g., Institute of Electrical andElectronic Engineers (IEEE®) 802.11, WiGIG, BlueTooth®, Zigbee®, NFC,etc.). As another example, the articles may each include a single set ofelements directly wired together; the display, the sensors, a smallprocessor, and a wireless interface. As disclosed the correspondingecosystem may include a smart phone with a smart mask (or article)application that is used to communicate with the control modules of thearticles via the wireless interface. The control modules may receive thesensor inputs, transform the inputs to display outputs, and then displaythe outputs on one or more displays. Further, some embodiments may existwithin a personal area network (PAN) where the various elements of theinventive technologies communicate with each other or with other devicesin and about a person. In some embodiments, a cell phone or other mobiledevice can operate as a central hub of the PAN to coordinate activitiesamong the connected devices including the smart article. An example PANsystem that can be leveraged for use with the disclosure subject matterare described in U.S. Pat. No. 10,667,212.

The ecosystem may include, in addition to the articles, othercomponents. For example, the ecosystem may include devices (e.g., thecentral control station 102 of FIG. 1) with software applicationssupporting management of the articles. The software may include featuresfor creating training data sets, capturing personalized images of aperson's face, downloading data (e.g., machine learning code, images,etc.) to the articles, updating software instructions on the articles,and other management capabilities. Additional management capabilitiesinclude monitoring articles, inventorying assets of the articles (e.g.,software version number, machine learning code version, digital imageassets, etc.), logging events, generating alerts, generating reports,recovering a crashed device, securing assets possibly via cryptographictechniques, and/or other features. The software may be configured toexecute on any suitable computing platform. The central control stationmay thus be, for example, a desktop computer, game console, smart phone,or other computing device. The software may be implemented on the mobiledevice 106 (e.g., smart phone, tablet, laptop computer, etc.).

The sensors 212, 412 of FIGS. 2 and 4 may include various sensors invarious arrangements. The sensors may include piezoelectric strain orstress sensors that generate voltages based on a stress or strain. Thesensors may be placed across a mask, face and/or other locations fromwhich data may be taken. For example, stress sensors may be placed alongstrings (or bands) of a mask and/or across a surface of the mask and/orbe used to connect the mask to the face. In an embodiment, the voltagesare detected and converted to sensor data. This may be done by ananalog-to-digital converter providing a one byte value (i.e., 0 to 255)or other digital forms; e.g., two bytes, three bytes, integers, floatingpoint values, etc. Detecting a voltage based on stress represents asingle sensed data modality. Other types of sensors may be included todetect addition data modalities. The sensors may include: thermalsensors to detect changes in emotion via temperature; galvanic sensors,capacitance sensors, resistance sensors, pressure sensors, and/or HallEffect sensors to detect motion and/or position; airflow sensors todetect breath; accelerometers to detect head movement; gyroscopicsensors to detect head movement; audio sensors to detect spoken works orutterances; and/or other types of sensors.

The sensors may be placed in different arrangements that meet the needsof a predetermined article complexity and use of the article. Forexample, stretch or strain sensors may be placed vertically (i.e., aline parallel to the nose to chin line). FIG. 5A shows an example smartmask 500 including a single stretch sensor 502 in a similar arrangement.The sensor 502 is connected to a control circuit 504, which may includeitems in the smart mask visual communication system 402, such as thecontrol module 404, the display 406, the memory 408, and the transceiver410. The stretch sensor 502 may be used to detect movement of a person'sjaw as the person speaks. As another example, three sensors may beplaced about a mask; one placed half way between a right ear and rightcorner of a mouth, one placed in a middle of a face running from nose tochin, and another placed half way between a left ear and left corner ofthe mouth. Each signature of the sensors and/or a signature of thecombined sensors aids in differentiating various jaw movements, such asspeaking, emoting, grinding teeth, left or right shifting of the jaw,and other movements. Still further, a stress or strain sensor may beplaced on a mask to be across a bridge of a nose to detect movement of amuscle around the nose. For example, if a person wrinkles the person'snose, possibly in disgust, the sensor would detect movements of musclesaround the nose and might display a frown or other image representingthe emotion of disgust. Air flow sensors can be used to detect inhalingor exhaling that might correspond to a gasp, cough, or other expressionsthat can translated to a corresponding or desirable image.

The sensors may also be directly coupled to a face of a person wearing acorresponding mask. Adhesive sensors may be placed beneath the mask,near the mask, and/or away from the mask. The sensors may be coupled toa control module via wired leads. For example, adhesive sensors may beplaced near outer corners of eyes, near nostrils, near ears, and/orother locations. The sensors may detect small movement of muscles atthese locations, which may correspond to emotions of the person, such aswhen a person smiles, but does not open the person's mouth. Often, theskin near the eyes wrinkle bringing life to the smile, which may bedetected by such adhesive sensors. The sensors may include flexiblesensors such as: bendable sensors; flexible touch force sensors;piezoelectric strip sensors; conductive rubber stretch sensors; stretch,splay, and compression sensors; adhesive thermal sensors; printedflexible sensors; Hall Effect sensors; etc. Example bendable sensors maybe found at URL www.bendlabs.com/products, a flexible touch force sensormay be found at URLwww.tekscan.com/products-solutions/electronics/flexiforce-oem-development-kit,a piezoelectric strip sensor may be found at URLwww.omegapiezo.com/strip-actuators-bimorph-equivalent, a conductiverubber stretch sensor may be found at URL www.adafruit.com/product/519,stretch, splay, and compression sensors may be found at URLstretchsense.com/sensors, an adhesive thermal sensor may be found at URLstarboardmedical.com/skin-sensors, printed, flexible sensors may befound at URL www.conductivetech.com/products/flexible-printed-circuitry,and Hall Effect sensors may be found at URLwww.adafruit.com/product/158.

The sensors may include multi-purpose sensors, such as sensors or sensormaterial offering antimicrobial features. For example, the sensors mayinclude graphene-based nanomaterials having antibacterial features. Forexample, graphene-based nanomaterials have been found to offerantibacterial features (for example, see URLwww.ncbi.nlm.nih.gov/pmc/articles/PMC6567318/. Example graphene-basedsensors include: nanotube sensors (see URLwww.ncbi.nlm.nih.gov/pmc/articles/PMC6523954/pdf/nanomaterials-09-00496.pdf),graphene health sensors (see URLwww.frontiersin.org/articles/10.3389/fchem.2019.00399/full), grapheneflex sensors (see URL www.cheaptubes.com/graphene-sensors), andprintable graphene sensors (see URLwww.nature.com/articles/s41528-019-0061-5). See URLwww.sciencedirect.com/science/article/abs/pii/S0165993617300031 for anoverview of graphene sensors. See URL www.directa-plus.com/ for a sourceof graphene. An example of a graphene-based antibacterial mask isdescribed at URLwww.dezeen.com/2020/03/06/guardian-g-volt-face-mask-graphene-coronavirus-bacteria/.As an example, the materials of a mask and/or sensors of the mask may beinclude graphene-based antibacterial materials. The mask may includegraphene-based antibacterial material for filtering purposes. The maskmay include an array of graphene-based antibacterial based sensors fordetecting face movements. Similarly, copper also has antimicrobialproperties and may also be used as a foundation for at least some of thesensors. As another example, a mask may include layers of materialand/or sensors that are at least partially formed of copper for sensingand filtering purposes. Sensors based on silver would also be acceptableas it also has antimicrobial activity. An example of a mask includingone or more sensors and one or more filters is shown in FIG. 15.

The displays (e.g., the displays 206, 406 of FIGS. 2 and 4) of the smartarticles may be in various forms. In one embodiment, one or more of thesmart articles may not include a display. The displays may includedigital displays. In an example embodiment, a smart mask may beconfigured to render a visual digital image on the mask, where thedisplay is flexible. The digital display of the mask includes a flexibledisplay that can move or flex along with underlying facial movements.The displays may vary from low resolution displays to high resolutiondisplays.

A low-resolution display may include a sufficient number of LEDs topermit forming a simulation of a mouth. For example, a set of red (orother colors) LEDs arranged in a 20×5 grid may be used to render ormimic just mouth movements where the LEDs may be sewn directly into themask. The LEDs may be individually sewn into the mask or smart articleor may be arranged on a flexible material layer of the mask and/or smartarticle. This “grid” may have a reduced number of LEDs by reducing thenumber of LEDs positioned near the corner of a mouth because the mouthcorners would not move much during speech. The array of LEDs may beconnected via flexible wires to permit movement and airflow. In variousimplementations, the grid of LEDs may be considered low-resolution ifhas a dots per inch (DPI) less than 70.

A high-resolution display may have sufficient resolution to display afull graphical image of a person's face and/or other images. In variousimplementations, the grid of LEDs may be considered high-resolution ifhas a dpi (dots per inch) greater than 70. The display may be a highdefinition (HD) or 4K (horizontal resolution of approximately 4,000pixels, pixel density of 550 pixels per inch (PPI), 880 PPI, or evenhigher) display. The displays may include and/or be communicativelycoupled to a processor, a power source (e.g., batteries, electricalcord, etc.), a graphics interface, and/or graphics processing unit(GPU). In other embodiments, the display 406 may be replaced with one ormore screens, such as one or more green screens or projector screens onwhich images may be displayed as further described below. Further,display 406 can include a composite display including multiple smallerdisplays working in concert. Individually sewable LEDs can be found atURL www.adafruit.com/product/1755. A 160×32 display can be found at URLwww.crystalfontz.com/product/cfal16032a0018pw-flexible-oled-display-160x32.An HD display can be found at URLnewvisiondisplay.com/6-flexible-amoled/.

The mask 500 may include bands 505 that extend around, for example, aperson's ears or back of the head. The bands 505 may be formed ofelastic material and be stretchable or have a fixed length. The bands505 may be replaced with ties and/or other attachment members. Similarbands are shown in FIGS. 5B-8.

FIGS. 5B, 5C and 5D show the smart mask 500 including LEDs (a couple ofwhich are designated 506). Although a particular arrangement of LEDs areshown including a particular number of LEDs, the LEDs may be in adifferent arrangement and include a different number of LEDs. In FIG.5B, the smart mask is shown with the LEDs in an OFF state. In FIG. 5C,some of the LEDs (a couple of which are designated 510) are shown in anON state due to a person's mouth being in a closed state where the ONLEDs form a line across the mouth region of the mask. In FIG. 5D, otherones of the LEDs (a couple of which are designated 512) are in an ONstate due to a person's mouth being in an open state.

FIG. 6 shows a smart mask 600 including multiple flexible displays 602,604. A mask may include more than one display, as shown. For example,the mask 600 may have a seam 606 joining a left and right portion of themask and that extends from a nose area 608 to a chin area 610. In suchcases, two or more flexible displays may be used to create a compound orcomposite display. For example, a single flexible display (display 602)may be positioned on the right side of a face and a second flexibledisplay (display 604) may be positioned on the left side of the face,where the two displays mate at the seam 606.

A display may be integrated with a smart mask in several ways. In someembodiments, the display is integral to the smart mask, while in otherembodiments, the display is part of a smart mask overlay that fits overan existing mask. Yet further, the displays may fit within transparentpockets to a mask system, which has advantages for modularity, upgrades,or repairs. Some examples of these embodiments are shown in FIGS. 7-15.Although not shown in these figures, the example, may includetransceivers connected to the shown control modules for communicatingwith other devices as described herein. The transceivers may be attachedto and/or integrated in any of the material layers shown.

FIG. 7 shows a smart mask 700 including a single flexible display 702 ina transparent pocket 704. FIG. 8 shows a smart mask 800 includingmultiple flexible displays 802, 804 in transparent pockets 806, 808.

FIG. 9 shows a portion 900 of a smart mask including an embedded display902 and sensors 904 with skin adhering contacts 906. The display 902 isembedded, connected to, and/or recessed in a first fabric (or material)layer 910. A power source 912 and a control module 914 are disposed inan intermediate fabric (or material) layer 916. The sensors 904 aredisposed in another fabric (or material) layer 918. The contacts 906 areconnected to the sensors 904 and may protrude outward from the fabriclayer 918. A centerline 920 is shown. The control module 914 may beconfigured similarly as the control module 404 of FIG. 4.

FIG. 10 shows a portion 1000 of a smart mask including a single stretchsensor 1002 and an overlay layer 1004 including a display 1006. Thedisplay 1006 is embedded, connected to, and/or recessed in the overlaylayer 1004. The overlay layer 1004 may be disposed on a firstintermediate fabric (or material) layer 1008. A power source 1012 and acontrol module 1014 are disposed in the first intermediate layer 1008 orin another intermediate fabric (or material) layer 1016, as shown. Thesensor 1002 is disposed in another fabric (or material) layer 1018. Acenterline 1020 is shown. The control module 1014 may be configuredsimilarly as the control module 404 of FIG. 4.

FIG. 11 shows a portion 1100 of a smart mask including multiple sensors1102 and an overlay layer 1104 including a display 1106. The display1106 is embedded, connected to, and/or recessed in the overlay layer1104. The overlay layer 1104 may be disposed on a first intermediatefabric (or material) layer 1108. A power source 1112 and a controlmodule 1114 are disposed in the first intermediate layer 1108 or inanother intermediate fabric (or material) layer 1116. The sensors 1102are disposed in another fabric (or material) layer 1118. A centerline1120 is shown. The control module 1114 may be configured similarly asthe control module 404 of FIG. 4.

FIG. 12 shows a portion 1200 of a smart mask including an overlay layer1202, a sensor 1204 and a perforated display 1206. The display 1206 isembedded, connected to, and/or recessed in the overlay layer 1202 andincludes perforations (or holes) 1207. The overlay layer 1202 may bedisposed on a first intermediate layer 1208 including spacers 1209. Apower source 1212 and a control module 1214 are disposed in a secondintermediate (or fabric) layer 1216. The sensor 1204 is disposed inanother fabric (or material) layer 1218. A centerline 1220 is shown. Thecontrol module 1214 may be configured similarly as the control module404 of FIG. 4.

FIG. 13 shows a portion 1300 of a smart mask including multiple sensors1302 and an overlay layer 1304 with a display 1306 spaced away fromother layers via spacers 1307 and/or a spacing layer 1308. The display1306 is embedded, connected to, and/or recessed in the overlay layer1304. The overlay layer 1304 may be disposed on a first intermediatefabric (or material) layer 1309. A power source 1312 and a controlmodule 1314 are disposed in the first intermediate layer 1309 or inanother intermediate fabric (or material) layer 1316. The sensors 1302are disposed in another fabric (or material) layer 1318 and may beconnected to adhesive contacts 1319. A centerline 1320 is shown. Thecontrol module 1314 may be configured similarly as the control module404 of FIG. 4.

The displays of a smart article may be positioned on risers to create anair gap, air plenums, and/or air channels through which air can flowfreely under the displays. See, for example, FIG. 14, which shows aportion 1400 of a smart mask including multiple sensors 1402 and acooling layer 1404 with spacers 1405 and channels 1406 for cooling adisplay 1407. The display 1407 is embedded, connected to, and/orrecessed in an overlay layer 1408.

The cooling layer 1404 may be disposed on a first intermediate fabric(or material) layer 1409. A power source 1412 and a control module 1414are disposed in the first intermediate layer 1409 or in anotherintermediate fabric (or material) layer 1416. The sensors 1402 aredisposed in another fabric (or material) layer 1418 and may be connectedto adhesive contacts 1419. A centerline 1420 is shown. The controlmodule 1414 may be configured similarly as the control module 404 ofFIG. 4.

FIG. 15 shows a face mask 1500 including a filter 1502 with embeddedsensors 1504. The filter 1502 may include a stack of porous sheets ofmaterial (e.g., graphene, copper mesh, silver mesh, etc.) that permitairflow and have electrical properties that change and are measurable inreal-time and operate as a sensor (e.g., stress, strain, etc. typesensor). Other configurations are also possible. For example, thesensors and filter material may be layered on one or more substratesthat are porous or have air channels for airflow. An example poroussubstrate 1506 is shown having holes 1508. In the example shown, thefilter 1502 is disposed in a pocket 1510.

In some embodiments, the flexible displays of the examples disclosedherein permit airflow through the corresponding masks. A display mayhave a mesh and/or a substrate with a set of holes that permits airflowthrough the display.

Although each of the masks of FIGS. 5A-15 include certain features, eachof the masks may include any of the features shown in FIGS. 5A-15 and/orotherwise disclosed herein. Similarly, although each of the masks ofFIGS. 5A-15 are shown as not including certain features that are shownin other ones of FIGS. 5A-15 and/or disclosed herein, each of the masksmay include any of the features shown in FIGS. 5A-15 and/or disclosedherein.

From a high-level perspective, a mask controller receives and/orotherwise obtains sensor inputs and converts the inputs to output valuesrepresenting facial positions or other types of classifications. Theoutput values are then used to determine what images are to be renderedon the display of the mask. More specifically and in some embodiments,the sensed values are compiled as an input vector and fed into (orprovided as inputs to) one or more functions that convert the sensedvalues to facial landmark positions or classes. The landmark positionsmay be used to generate corresponding rendered images of the person'sface and facial features at positions corresponding to landmarkpositions. Example landmark positions are shown in FIGS. 16B-D.

FIG. 16A shows a mouth of a person in a silent and neutral position.FIG. 16B shows the mouth of FIG. 16A with overlaid landmarks and anorigin. The origin is a reference point, which may be centrally locatedin an image or located elsewhere. FIG. 16C shows the mouth with theoverlaid landmarks moved to different locations associated with mouthmovement (or an open mouth position) and relative to the origin. FIG.16D illustrates positioning of landmarks relative to the origin.

For illustrative purposes, consider a scenario where a mask includesfour sensors that generate values between 0 and 255. At any given sensedtime (e.g., periodically, when triggered by activation criteria, everymillisecond (ms), every 5 ms, etc.), the control module of the mask maygenerate an input vector of four bytes, where each byte represents avalue of a corresponding sensor. Four facial landmarks may beimplemented: two landmarks over respective corners of the mouth, oneover the upper lip, and another over the low lip of the person. The fourbyte sensor input vector is mapped and converted to a set of positionsfor the four landmarks (e.g., (x, y) coordinates for each landmark, asingle relative distance value from an origin, and/or other positionvalue, etc.). In this example, the mapping function does may not beone-to-one (i.e., one sensor value corresponds to a single facefeature). Rather, the four sensor inputs may be indices of a 4D space,which yields a corresponding landmark position. The 4D space cansub-divided into regions where reach region represents mouth positionsor landmark positions. This may be referred to as a brute force, look-upapproach. The mapping space may be reduced by using principle componentanalysis to achieve a target, more memory efficient, output mapping.

In this example, the mapping function may operate as a classifierimplementing a support vector machine (SVM) algorithm, a k nearestneighbor (kNN) search algorithm, a random forest algorithm, etc., wherethe landmark positions represent classes. In such an approach, theidentified classes may correspond to specific images that should bedisplayed. Thus, the mapping function or functions for the fourlandmarks may be at least similar to the following: Rc(S), Lc(S), UL(S),LL(S) where Rc==right corner of mouth; Lc==left corner of mouth;UL==upper lip position of mouth; LL==lower lip position of mouth. S isan input vector having sensor values. One should keep in mind that thesemapping functions represent software instructions and may thus employvarious executable instructions stored in memory. As mentionedpreviously, the mapping functions may leverage an implementation of aclassification algorithm. However, the mapping functions may also beimplemented as a lookup table as alluded to above.

AI Training

In one embodiment, artificial intelligence (AI) training is implemented.This may be done to train any of the control modules referred to hereinincluding the modules 110, 204, 302, 404 of FIGS. 1-4. The AI trainingmay include any of the below described training features and operationsincluding operations similar to that performed during the methods ofFIGS. 17-19. The operations of FIGS. 17-19 may be performed subsequentto and/or while AI training is being performed.

According to an example embodiment, an AI visual data training set maybe created by recording a video of a person reading a preset text thatcauses the person's face, without wearing a mask, to make knownmovements as matched with spoken words and/or with known expressions(phonemes, smile, frown, smirk, emotes, utterances, etc.) This trainingset provides for capturing images of the person's face for laterrendering on one of the smart masks and/or articles disclosed herein. AnAI sensor data training set may be created by having a person put on adigital display mask and/or sensors and repeating the same trainingprocess as described above while recording sensor data, which may bereceived at and/or by one or more of the smart masks and/or articlesdisclosed herein. This training data set may be compiled at a same timeas the image training data set. The training data provides for matchingsensor data to the corresponding spoken words and to the correspondingimages of the person's face at various positions and/or duringmovements.

A neural network implemented by one of the control modules is trainedbased on the training data set such that sensor data input is convertedto a visual representation of the person's face. As the person speakswith the smart mask on, the AI senses the real-time sensor data, whichmay include audio (e.g., voice) data or other data modalities. The maskuses the real-time sensor data, which may include facial movement (orphysical face) data and/or voice data as input to the trained neuralnetwork, which outputs an identifier representing an image of theperson's face corresponding to the neural network inputs.

While one of the preceding examples focuses on converting the sensorinputs to four facial landmarks, the mapping function implemented by oneof the control modules may operate as an intermediary between the sensorinputs and the displayed images. Thus, the mapping function may simplygenerate an image identifier (ID) corresponding to an image to bedisplayed. For example, the image ID may be a file name of an imagestored in the memory (e.g., a file system) of the control module of themask.

As another example, 44 images may be generated to be displayed, eachimage corresponds to a mouth arrangement for 44 known phonemes inEnglish. The mapping function may generate an image ID of 37 indicatingthat image 37.png should be rendered. The approach of converting thesensor inputs to an image ID is considered advantageous because itallows the user to swap out the images and replace them with any otherimages (in some embodiments, as long as the images have file names inthe same ID space). Thus, a control module may replace facial images ofa first person with images from another person's face (e.g., acelebrity, a cartoon character, etc.) based on user input. In anotherembodiment, the images do not correspond to a face or phoneme mouthposition, but are images of something else, such as: a tree, a car, adog, a pet, a logo, a set of emoticons, etc. It should be appreciatedthat such embodiments might be best served as novelties. While it iscontemplated that the training software might map to known phonemes oremotions, it is also contemplated that a user could create customconfigurations where they map a sensor signature to a custom image. Forexample, a user might map non-phoneme utterances (e.g., “Ugh”, “Arg”,“Hmmm”, raspberry sound, etc.) or other facial movements to a desiredimage. Perhaps “Arg” might be mapped to an image of pirate, or anemoticon expressing anger.

In more complex implementations the number of sensors may vary and thenumber of face positions and/or landmarks may vary. The number of sensorinputs may not correspond to the number of face positions or landmarks.While some embodiments include one sensor state mapping to one outputimage, other embodiments are more complex and include generating morethan one output image. Thus, the mapping function may take on a fullspectrum of mappings: one-to-one, one-to-many, many-to-many, etc.

While the disclosure to this point may be directed to a personalized orcustomized mapping function, a mapping function may operate based on adefault configuration or a default setting. The default configurationmay be based on a generic mapping of sensor values to a set of genericmouth arrangements corresponding to spoken phonemes, where thearrangements may correspond to landmarks that are tracked. Further thedefault configuration may be compiled based on AI or machine learningtraining data sets compiled from many individuals. Example moutharrangements and positions may be mapped to corresponding phonemes. SeeURL www.rose-medical.com/mouth-positions.html.

Another example of a more complex embodiment includes implementationsexecuting code of the mapping function corresponding to a trained neuralnetwork (NN) that accepts the sensor input vector and outputs desiresmouth positions, landmark positions, image IDs, and/or other informationused to render a desired corresponding image. In NN embodiments, one ormore training data sets are compiled based on sensor values andcorresponding to spoken phonemes, emoted facial positions, custom sensorsignatures, and/or other facial arrangements. Further, in someembodiments, the training data set may be compiled on a person-by-personbasis to ensure the training data is highly personalized. In otherembodiments, as referenced above, the training data set may be builtfrom hundreds, thousands, or more users thus representing a default orbase configuration. The training data set could be created via acrowd-sourced effort where data is collected and integrated into thesoftware for the smart article ecosystem. An example technique thatcould be adapted for use in creating crowd-sourced face movement toimage training data sets are described in U.S. Pat. No. 10,147,038. Datafrom the specific user may then be captured followed by the NN trainedon the default data set being refined based on the user's data set. Inyet another embodiment, the training data set may be the default dataset without modification.

A mask management software application may be executed by one or more ofthe control modules to create a training data set. The mask managementsoftware may be executed to observe a person's face and/or soundsuttered by the person via a camera and/or microphone. Landmarks on theperson's face are observed while the person's face proceeds throughknown positions.

Corresponding images, face positions, landmark positions, and/or facialfeatures are captured and recorded. Facial features may be observedusing face recognition capabilities and face feature detectioncapabilities. Facial features can be observed based on one or moreavailable software packages, such as OpenCV (see URL www.opencv.org),which offers both face recognition capabilities and face featuredetection capability. For example, the application may use one or moreimplementations of feature detection algorithms to track the landmarksof a person's face. Example algorithms include Canny edge detection,edge vectors (see U.S. Pat. No. 9,412,176), scale-invariant featuretransform (SIFT; see U.S. Pat. No. 6,711,293), histogram of orientedgradients (HoG), speeded up robust features (SURF), or others. Further,the algorithms may track specific landmarks such as the 68 landmarks aspart of an intelligent behavior understanding group (iBUG) 300-W dataset(see URL ibug.doc.ic.ac.uk/resources/facial-point-annotations/).

Returning to compiling the training data set, each of the faciallandmarks may be tracked by the management application as a personenacts a known script, where the script corresponds to known facialpositions. The script may include a set of words that correspond tophonemes (see URL en.wikipedia.org/wiki/Phoneme), which have knownpositions. Further the script may include emotional actions such assmile with mouth open, smile with mouth closed, frown, laugh, wince,emote fear, emote surprise, etc. Naturally, every language has arespective set of phenomes. For example, English is considered to have13 to 21 vowel phonemes and about 22 to 26 consonant phonemes; around 44total. Thus, in English, the person may recite a script that has theperson utter each of the phonemes, preferably multiple times and indifferent orders to ensure capturing various aspects of the persons faceas the portions of the face move (e.g., still images of mouth, audio ofvoice, video of mouth, transitions from one phoneme to another, etc.).Thus, the software compiles a mapping of the phonemes to landmarks aswell as a mapping of the phonemes to actual images of the persons faceat the moment or the process of uttering the phonemes. A list of wordsthat correspond to 44 phonemes in English are known and may be found atURL www.dyslexia-reading-well.com/44-phonemes-in-english.html. The wordsfrom this list may be used to construct a script to be read by the user.In some embodiments, the user can create custom scripts and customimages to fit their own needs.

One advantage of having the person read multiple scripts havingdifferent orders of phonemes is the management application may identifytransitions of facial landmarks from one phoneme to another and therebycapture video of the transitions. Videos of such transitions are usedfor improved rendering fidelity and performance when the person wearsthe mask and speaks. The video transitions or a digitally constructedtransition (e.g., a 3D digital model of a face, a 2D animation model,etc.) may be rendered in real-time on the display of the smart mask (orarticle).

As the control module executing the management application observes aperson, the control module may display the script to prompt the personto say the specific words in the script at specific times. Whileobserving the person, the software may employ speech recognition to mapthe landmark tracking to the phonemes. Further, by presenting the promptto the person at the same time as employing the speech recognition, thesoftware is able to better handle accents.

FIGS. 16A-D (collectively FIG. 16) provide a simple illustration ofmapping landmarks of a person's mouth to a phoneme (see URLwww.rose-medical.com/mouth-positions.html). As shown, the mouth may beinitially at rest and thus in a silent or neutral positon. This imagemay be displayed on the mask when the sensors lack any sensed movement.While at rest or at a neutral position, the software executes one ormore implementations of facial detection algorithms (e.g., OpenCV, etc.)to identify landmarks in the image data. In the example show, a smallnumber of landmarks are shown relative to an origin point. The origin isshown as a location near the center of the lips, but may be anypractical location or might not be necessary in some implementations.Further, in some embodiments, the origin corresponds to the center ofthe mask's display or a position relative to the center of the mask'sdisplay.

The origin point may be found based on the landmarks or other imagefeatures. For example, a center point of the lips and/or a fixeddistance from the nose. Such positions may be calculated bytriangulation, calculating a centroid of landmarks, and/or othersuitable techniques. As a person speaks the phonemes, the landmarks movein space. In the example shown, the mouth is in a position to articulatethe “k” sound in “kit”. The facial feature detection algorithm tracksthe movement of the landmarks and calculates the new positions inreal-time or near real-time relative to the origin or relative to thelandmarks' previous positions. Thus, the software is executed togenerate data representing landmark positions and movement duringphoneme utterances.

In some embodiments, the data is a set coordinate values. For example,the coordinates could be (X, Y) coordinates relative to the origin,difference in X and Y relative to origin, difference in X and Y relativeto previously positions, or other measurements. While the origin isshown in the middle of the lips, the origin may be placed at anyposition that is able to be reliably determined. The origin may be usedto as a reference point for rendering the mouth or lip images on themask in a consistent manner. Such rendering may be achieved through agraphics rendering engine that moves a 2D or 3D model of a mouthrelative to the origin with textures from the person. However, an originis not necessary in embodiments that simply display static images of aperson's mouth that correspond to the sensed positions, sensedutterances, sensed emotes, or according to other sensor signatures.

Once the management software completes observation of the person readinga script, the software has a data set mapping the phonemes to faciallandmarks and/or facial image features. This mapping may also includeaudio data of uttered phonemes in the person's voice that also map thefacial landmarks or image features.

At this point, the software maps phonemes or utterances to a visualrendering. Further data would also be useful to create a mapping of masksensor data (e.g., facial movement, audio, etc.) to the facial landmarksor image features. A mask can obscure a person's face. Sensors are thusused to detect actual movement of the person's face beneath the mask todetermine what images or renderings to display via the display of themask.

The management software, the same application or another data collectionapplication, running on a data collection device (e.g., a personalcomputer, a cell phone, the mobile device of FIG. 3, etc.) creates adata set representing a mapping of the sensor data to the phonemesand/or facial landmarks as the person enacts or reads the script. Thedata set may be created by having the user wear the sensors associatedwith the mask (e.g., the mask itself if the sensors are embedded or justthe sensors if the sensors are not embedded at the same time as readingthe script to capture image data). The person then may repeat thescript. In embodiments where the sensors are separate from the mask anddo not obscure the face, sensor data collection process can be done atthe same time as the image data collection process. For the sake ofdiscussion, assume the sensors are integrated into the mask. The personagain enacts or reads the script by following prompts generated throughexecution of the software as the script is displayed to the user via adisplay screen. The software may include code for displaying the script.The software is executed to collect corresponding sensor informationoccurring at the same time as the person speaks the phonemes or emotes.The resulting data then includes a collection of sensor data vectors(e.g., a data set having one or more sensor data feeds) that map tocorresponding phonemes, landmarks, and facial image data. This processmay be repeated any number of times to collect larger data sets by whichthe system learns. Further, the training data may include data fromdozens, hundreds, thousands or more users.

The sensor data set may take on many different forms depending on thedesired complexity. In low complexity embodiments, the sensor data mayinclude static values obtained from the sensors when the managementsoftware determines the person's face is in a proper position. Forexample, the software might display the word “kit” for the person toread. The software may identify two main phonemes “k” and “t” for thesake of discussion. Leveraging audio data, the software may determinethe sensor data at the time the “k” sound was uttered and the sensordata at the time the “t” sound was uttered. Thus, in this simpleexample, two sensor data sets would be compiled: one for “k” and one for“t”. More specifically, if there are four strain sensors having valuesbetween 0 and 255, the “k” sound set would have four bytes specific tothe “k” sound and four byes specific “t”. In some phoneme scenarios, themouth positions may be very similar and the sensor data may not beconclusive as to the phoneme being uttered. In moderately complexembodiments, other sensor modalities may come into play. For example, anaudio transducer may be included that detects the sound of a person'svoice. Using another sensor data modality may offer advantages includingdifferentiating among similar sensor signatures. In yet more complexembodiments, the senor data may include time-series data representinghow the sensor data changes with time. Thus, the time-series sensor datamay be mapped directly to the time series image data of the person'sface (e.g., video, etc.).

The sensor data may include time series data. Some embodiments mayleverage values derived from the sensor data in lieu of or in additionto the sensor data. For example, derived values may include higher ordertime derivatives of the sensor data. Using derived values, which mayalso be time series, aids by mapping sensed mouth movements to movementof facial landmarks in the image data and further aids in identifyingtransitions from one phoneme to another. Both the sensor data and thefirst time derivative of the sensors may be used as an input. In thiscase, a vector may have a time value, sensor values, and timederivatives of the sensor data. Thus, for a four-sensor system, theinput vector would have nine values. If a transition cannot be detectedin real-time, the mask control can display a default transition imagesuch as a mouth at rest, then proceed to the next image as indicated bythe sensor data.

Returning to the use of machine learning techniques, one shouldappreciate that the disclosed approach could use classifier techniques,regression techniques and/or other machine learning techniques toconvert the sensor data to renderable images. Classifiers (e.g., SVM,kNN, NNs, etc.) may be trained based on the sensor training data setswith respect to a set of images corresponding to the phonemes. Thus, theclassifier accepts the sensor input and may generate at least one image,which is then displayed on the mask's display. In this embodiment, theset of images may include just static images for the various phonemes,emotes, or other mouth positions, where the static images represent theclasses for the classifier. Further, each static image can be displayedfor a desired amount of time before displaying a next image (e.g.,display for 0.2 seconds, display until a next image is found, etc.).While these examples are presented with respect to classes representedby images, the classes may also be defined by positions of faciallandmarks, which may then be used to generate a graphical 2D renderingof the person's mouth possibly based on a digital model of a mouth orface.

It is also possible to combine multiple classifiers together, an SVMplus an artificial neural network (ANN) for example, where the resultsof multiple classifiers represent a “vote” for an image to be displayed.Regression techniques offer greater fidelity when displaying imagesbecause regression techniques generate interpolations between or amongvarious positions. Rather than generating a classification, a regressiontechnique generates predictive or forecasted values representingpositions of the mouth as a function of the sensor data. Such techniquesconvert sensor time-series data to corresponding mouth positions orfacial landmark positions or movements as time-series data. There aremany types of machine learning regression models (e.g., linearregression, ridge regression, least absolute shrinkage and selectionoperator (LASSO) regression, elastic net regression, neural networks,etc.), from which to choose. The choice of one or more regression modelsdepends on the desired complexity of the resulting displayed images andthe number of sensors in the system.

As discussed above, the sensor data may be mapped to facial landmarks,especially mouth landmarks. For example, the executed software may tracksuch landmarks. For example, computing packages, including dlib orOpenCV, offer the ability to track such landmarks. As referenced furtherabove, the facial landmarks may include 68 landmarks, where 20represents the mouth (see URLibug.doc.ic.ac.uk/resources/facial-point-annotations/). The outputs ofthe sensors may be mapped to the corresponding positions and/or movementof the landmarks. As another example, four sensors and 20 landmarks maybe monitored.

The 20 landmarks may each have an (x,y) coordinate, which may be trackedrelative to an origin based on the phonemes or emotes as sensed by thesensors. Assuming for the sake of discussion, the outputs of the foursensors are mapped to the 20 landmarks, such that the input vector tothe neural network is four bytes. The output of the neural network mayinclude 20 (x,y) value pairs corresponding to the predicted positions ofthe 20 mouth landmarks in a suitable coordinate system for display,where the prediction positions may then be used to create a rendering ofthe person's mouth on the mask's display. Multiple differentarchitectures may be implemented for a neural network that providesthese features, all of which are considered to fall within the scope ofthis disclosure. The complexity of the neural network may vary dependingon the fidelity of the output, cost to build, and/or other factors.

The NNs may leverage a recurrent neural network (RNN) that receives fourinputs and generates 20 output values. In general, NNs that have reducedmemory footprints and quickly generate a result in real-time reduce thecost to create the mask. For example, in the instant example, the NN mayuse an input layer having four nodes, one or more fully connected hiddenlayers, and an output layer having 20 to 40 nodes depending on thedesired output, where the output layers provide values that may bemapped to facial landmarks. For such a use case, a convolutional neuralnetwork may prove useful as it would have a smaller memory foot print.The type of model used to map the sensor outputs to the displayedoutputs may be varied to fit a particular need. Thus, the model used maybe a fully connected neural network, convolutional neural network, atransformer, a classifier, and/or other types of models.

As another example, a basic classifier of one of the control modulesdisclosed herein may be used for a mask with a single stretch sensor (anexample of which shown in FIG. 5A) stretching across the mask from noseto jaw, which fits a face snuggly. The display of the mask may include aset of sewn in LEDs arranged to mimic an open and closed mouth of theface. This mask is an example of a low-end, low cost embodiment. Thesensor has a resistance depending on a degree of stretch (e.g., highresistance when relaxed, low resistance when stretched, etc.). As thesensor stretches during mouth movement, the control module of the maskmay detect changes in the resistance (e.g., via a voltage change, viacurrent change, etc.). The control module may determine if the mouth isopen or closed. When the control module determines the mouth is closedvia a sensed resistance value or range, the control module turns theLEDs on in the closed arrangement. When the control module determinesthe mouth is opened via a different resistance value or range, thecontrol module turns off the LEDs in the closed arrangement and turns onthe LEDs in the open arrangement. The purpose of providing this simpleexample is to illustrate that the model used to determine mouth positionmay be a very basic classifier where sensor input values are compared tothresholds or a single criterion to determine a class (i.e., open orclosed) and display a corresponding result. Higher end masks may employmore sophisticated classifiers and/or regression models as disclosedherein having dozens, 100s, or more classes.

The systems and devices disclosed herein may be operated using numerousmethods, example methods are illustrated in FIGS. 17-19. The followingmethods may each include an AI and/or neural network learning process asdescribed herein, which may be implemented prior to and/or during any ofthe following methods.

In FIG. 17, an example article operation method is shown. Although thefollowing methods are shown as separate methods, the methods and/oroperations may be combined and performed as a single method. Althoughthe following operations are primarily described with respect to theimplementations of FIGS. 1-4, the operations may be easily modified toapply to other implementations of the present disclosure. Although FIG.17 is primarily described with respect to the smart article 400 of FIG.4, the operations are applicable to the smart mask 200 of FIG. 2. Theoperations may be iteratively performed.

The method of FIG. 17 begins at 1702, where the article control module404 may be powered ON and initiate facial movement tracking softwarestored in the memory 408 and executed by the article control module 404.

At 1704, the transceiver 410 may establish a connection with one or moredevices, such as the central control station 102 and/or the mobiledevice 300. At 1706, the article control module 404 may collect sensordata from the sensors 412. At 1708, the article control module 404 maytransmit the collected sensor data to the central control station 102and/or the mobile device 300.

At 1710, the article control module 404 may receive mapped devicecompatible data and/or display signals from the central control station102 and/or the mobile device 300 based on the collected sensor data. At1712, the article control module 404 may convert the mapped devicecompatible data to display signals and display images based on themapped device compatible data or display images based on the receiveddisplay signals. At 1714, the article control module 404 may receivesensor data from one or more other smart articles. At 1716, the mappingmodule 415 may map the sensor data collected from the sensors 412 and/orfrom the other one or more smart articles to mapped device compatibledata and/or display signals.

At 1718, the article control module 404 may transmit the mapped devicecompatible data and/or display signals to the other one or more smartarticles. Operation 1712 may be performed subsequent to operation 1716and display the mapped device compatible data and/or the display signalsgenerated at 1716. The method may end subsequent to operation 1712 ormay return to operation 1706 as shown.

FIG. 18 shows an example mobile device operation method. The methodbegins at 1802, where the smart article application 310 is started. Thismay be based on an input received from a user of the mobile device 300or other sensed triggering event.

At 1804, the mobile device control module 302 via the transceiver 308establishes connection(s) with one or more smart articles (e.g., thesmart articles 200 and 400) and may also establish a connection with thecentral control station 102. At 1806, the mobile device control module302 may receive sensor data from the one or more smart articles.

At 1808, the mapping module 311 may map the received sensor data todevice compatible data and/or display signals. At 1810, the mobiledevice control module 302 may transmit the collected data to the centralcontrol station 102.

At 1812, the mobile device control module 302 may receive mapped devicecompatible data and/or display signals from the central control stationbased on the transmitted collected data. At 1814, the mobile devicecontrol module 302 may transmit the mapped device compatible data and/ordisplay signals generated and/or received to the one or more smartarticles. The method may end subsequent to operation 1814 or may returnto operation 1806 as shown.

FIG. 19 shows an example central control station operation method. Themethod begins at 1902, where the central control module 110 of thecentral control station 102 may establish connection(s) with the mobiledevice 300 and/or one or more of the smart articles (e.g., the smartarticles 200 and 400).

At 1904, the transceiver 112 may receive sensor data from the sensors116, the mobile device 300 and/or the one or more smart articles. At1906, the central control module 110 may map the sensor data to mappeddevice compatible data and/or display signals. At 1908, the centralcontrol module 110 may transmit the mapped device compatible data and/ordisplay signals to the mobile device 300 and/or the one or more smartarticles. The method may end subsequent to operation 1908 or may returnto operation 1902 as shown.

The above-described operations of FIGS. 17-19 are meant to beillustrative examples. The operations may be performed sequentially,synchronously, simultaneously, continuously, during overlapping timeperiods or in a different order depending upon the application. Also,any of the operations may not be performed or skipped depending on theimplementation and/or sequence of events. Additional operations may alsobe performed as disclosed herein.

The following examples include additional details and variations of theabove-described embodiments. The operations described below may beimplemented by the smart articles disclosed herein.

Sensors may be arranged around a mask, face, mouth, or head of a maskwearer in various patterns and in various configurations. For example,stretch sensors may be placed across the mask diagonally from a lowerleft jaw to an upper right cheek and vice versa. Sensors may beintegrated into a mask, where the mask is properly fitted to a person'sface to ensure optimal sensor performance. A mask system may havevarious capabilities. Low-cost masks may have a few sensors. High-cost,high performance masks may have a larger number of high fidelitysensors. Additionally, sensors may be affixed to various articles arounda person including glasses, hats, scarves, shirt collars, and/or otherarticles and/or portions thereof. For example, image sensors may bedisposed near eyes to observe pupil dilation to determine emotionalresponse, which may then be used to determine which emotion to render ona display of a smart article or may be used to render a correspondingemotion.

As sensor complexity may vary, display functionality may also vary. Atone end of a spectrum, the displayed images on the mask do notnecessarily correspond to the actual phonemes, emotions, or otherexpressions. This end of the spectrum may be considered as a novelty forentertainment purposes, where the displayed images may be consideredcomical, entertaining, and/or just fun. At the other end of thespectrum, the precision of the displayed images on the mask matchclosely to actual expressions of the wearer. The high precision versionof the mask likely costs more, but is considered advantageous wherecommunication among individuals is critical. Example use cases includeinteractions with patients at points of care, business meetings, and/orother similar situations. An intermediary point on the spectrum betweenthe low-end and high-end may be leveraged by retail sales, where someprecision is necessary to convey interest in another person, but is notcritical.

While some embodiments as described may be based on an assumption thatdisplayed features correspond to sensed mouth positions, there is norequirement that the displayed images are images of a mouth. Forexample, the images displayed may include sign language, text in anative language of the speaker wearing the smart mask, translated textin another language, emoticons, and/or other images. Additionally, thedisplayed images may not be displayed on a mask, but may be displayed onother articles, such as a hat, shirt, a patch, and/or other connecteddisplay-enabled article. This may include smart articles as describedabove and/or other articles.

The images rendered on a display may be part of a video or may be stillimages. In a video format, the rendered images may be presented inreal-time as the sensed data is received. For example, in someembodiments, a digital three-dimensional (3D) model of a person's facemay be built and then rendered for display based on the sensed datawhile using textures generated from images of the person's face. In astill image format, once an image is selected for display, possibly viaa classification technique as described above, the image may bedisplayed for a period of time commensurate to a corresponding utteranceor emotion. A current image may be replaced by fading into a next imagefor display thereby simulating a transition. Further, images may bemerged or interpolated to address a need for showing a person's mouthmove while smiling or frowning.

In other embodiments, the mask display may not be a physical displayintegrated as part of a mask or disposed on the mask wearer. Forexample, consider a newscaster. A field reporter may wear a green screenmask (i.e., a mask that is just green). As video of the reporter iscaptured and corresponding sensed data is received, the system of theexample embodiment may generate corresponding digital images (e.g.,chroma content, etc.) that are superimposed on the green mask therebyrendering the reporter's face for broadcast. Another example may includeusing a projector system that projects images on the mask of a liveactor. As an example, the central control station 102 of FIG. 1 and/orother device may be implemented as a projector and project orsuperimpose images on a projector screen or a green screen of one ormore smart articles (e.g., the smart article 400 of FIG. 4). Stillfurther, the display may be coupled with a robot, possibly atelepresence interface, where the robot displays a face corresponding toa controller.

The disclosed mask system examples may also be used with respect to facerecognition. The displayed images of a person's face may be augmented toinclude additional features that aid in recognizing a person's face,such additional features may include stronger contrast on key facialfeatures or descriptors to facilitate analysis by other computingsoftware. The reverse also has utility. The displayed images may beaugmented to reduce or disrupt facial recognition software. For example,the displayed images may correspond to cartoon versions of a person'sface or may de-enhance recognition features of the person's face therebyincreasing privacy of the mask wearer.

In some industries, face recognition technologies are used to identify aperson based on features in an image: eyes, nose, mouth, etc. Thedisclosed approach seeks to leverage these types of known techniques tocreate a collection of images of a person for display on a mask.

The movie industry leverages motion capture where actors wear greensuits or wear identifiable markers. As the actor moves, the markers aretracked. A digital avatar may be manipulated to move in the same way asthe actor by mapping the avatar's movements to the tracked markers.Further, digital or graphic renderings may be superimposed on theactor's bodies using traditional green screen techniques. Similar to theface recognition, tracking markers may be leveraged to create a mappingbetween movements of a person's face and a corresponding rendering.

Synchronizing a rendering of lips of video game characters to dialogshas been used by video games to create a more realistic experience forgame players. For example, U.S. Pat. No. 6,307,576 to Rosenfelddescribes such techniques. However, the example rendering of lips asdisclosed herein are two-dimensional (2D) constructs on articles.

As another example, sensed movement and/or other modalities may becombined to create a rendering on a mask. The masks may include graphenefilters and graphene sensors. The masks may include graphene forprotection and filtering, as well as for sensing, for example,movements. Graphene may be implemented as part of sensor material. Whilegraphene may be used as a biosensor, graphene's electrical propertiesallow graphene to be used in sensors disclosed herein to measure stressand/or strain. When graphene sensors are strained, the sensors generatean electrical signal that is amplified and detected. In one embodiment,a graphene material is not used as both a sensor and a filter at a sametime. The same graphene material may be used as a sensor or a filter atany instance in time. Different graphene material layers may be providedto perform respectively as a sensor and a filter.

Sensors attached to the face may be used to determine aspects of aperson's emotions. For example, graphene sensors may be used todetermine emotions, which may include stretchable and/or flexiblesensors. Facial movement may be detected in real-time using thesesensors.

Some traditional masks are clear such that a person's face can be seenwhile wearing one of the masks. However, such masks are not flexible.The examples provided herein provide masks that may be flexible andallow others to see facial expressions on a display and/or elsewhere asdescribed above.

The face masks and other articles disclosed herein may be used forvarious purposes and in various implementations. This includes use ofsmart masks, during, for example, a COVID pandemic. Other examples arereferred to below and include military, police/fire, motorcycle,theater, movie, medical, novelty, religion, sports, service industries,hearing impaired, autism, new casting, and computer game exampleapplications.

The military often is called upon to wear masks while interacting withcivilians. The disclosed techniques may be used to soften the appearanceof military personnel and make them more accessible to civilians. Apolice use case largely follows the military use case.

Many motorcyclists wear full head covering helmets. The disclosedtechniques may provide for displaying (or projecting) a cyclist'sexpressions on a surface of a helmet. As yet another example, actors mayleverage a purpose built face mask that covers portions or an entireface such that an actor may appear as another person or appear withdynamic makeup.

Similar to the theater example, during motion capture sessions where aperson wears a motion capture suit, the person's expressions may becaptured and mapped to a predetermined feature set using the examplesdisclosed herein.

As another example, a medical healthcare professional may implement someof the examples disclosed herein. As an example, high fidelity images ofthe professional may be taken and provided to aid in easing a patient'sexperience.

Yet another use case (novelty case) may include creating purpose builtHoliday (e.g., Halloween) masks having desired, novelty driven features.Further, one may monetize the disclosed techniques by selling noveltyanimations or perhaps celebrity images for a fee or subscription. As afurther example, the techniques may be applied to, for example, a burqaor niqab if permissible.

The examples may also be applied to helmets used in full helmet sports,such as American football. Aspects of a player's face may be displayedon a helmet. The displays on the helmets may be used for controllingparts of the game. Services industries such as restaurants would likelybenefit from the disclosed masks and articles to allow people to feelmore comfortable during dining experiences.

People interacting with the hearing impaired may wear the disclosedmasks and/or articles. The hearing-impaired person may see the person'smouth on one of the displays move in a more exaggerated manner thanactually being moved to improve lip reading. The mask may display textcorresponding to the person's speech. The mask may display an emoticonto match a person's mood and/or other emotions. Further, a hearingimpaired person could use a set of gloves having sensors as discussedabove with respect to the mask. As the hearing impaired person usestheir hands to communicate via sign language, their mask could displayimages representing mouth positions for the words they are signing. Yetfurther, the mask could be instrumented to generate corresponding audiosignals via a speaker where the audio signals comprise a digital voicespeaking the corresponding words. Similar to the hearing-impairedapproach, a mask may be used to display a person's emotional state suchthat people with autism are able to quickly interpret another person'semotion. Alternatively, a person could use the disclosed system topractice emoting to ensure they are properly providing social cues. Oneshould appreciate that the smart mask could display more than one imagemodality. For example, the mask could display a person's mouthmovements, corresponding text, emoticons, or other such features at thesame time.

During the COVID-19 pandemic, many newscasters have worn masks whilereporting to the public. The disclosed examples may be used to display anewscaster's mouth as the newscaster provides televised reports. Thesmart mask may also operate as a game controller where sensor data isused to determine control over gaming elements. Further, the smart maskmay be used in virtual reality (VR) systems, where a person's facialmovements are rendered in the VR setting on the wearer's VR avatar. Insuch embodiments, the mask technology disclosed herein can be integratedinto a VR headset, coupled to the VR headset, or coupled to the VRcomputing system. In such an arrangement, a VR user could have theiravatars displayed with the user's current mouth positions.

Yet another aspect of the inventive technology includes leveraging alocation sensor (e.g., GPS sensor, internal movement unit, SLAM, vSLAM,wireless triangulation device, etc.) to determine a location of thesmart article. Location data can be leveraged to select one or morelanguage modules for translation purposes. For example, if a personspeaks English, but is wearing a mask in Italy, the displayed imagescould include Italian translated text. U.S. Pat. No. 10,367,652describes location-based techniques for selecting device interactiondomains that can be adapted for use with the inventive subject matter.Further, location could be used to select a desired image set. If aperson is located in an informal setting (e.g., at a park, ballgame,etc.), the displayed images of mouth movements might corresponding to apurchased images of a favorite cartoon character, while when the personis located in a formal setting (e.g., an office, etc.), the imagesselected for display might include the person's actual face images. Thiscan be achieved via a look up table, mapping the location data to ageo-fenced area, or other form of indexing.

CONCLUSION

The foregoing description is merely illustrative in nature and is in noway intended to limit the disclosure, its application, or uses. Thebroad teachings of the disclosure can be implemented in a variety offorms. Therefore, while this disclosure includes particular examples,the true scope of the disclosure should not be so limited since othermodifications will become apparent upon a study of the drawings, thespecification, and the following claims. It should be understood thatone or more steps within a method may be executed in different order (orconcurrently) without altering the principles of the present disclosure.Further, although each of the embodiments is described above as havingcertain features, any one or more of those features described withrespect to any embodiment of the disclosure can be implemented in and/orcombined with features of any of the other embodiments, even if thatcombination is not explicitly described. In other words, the describedembodiments are not mutually exclusive, and permutations of one or moreembodiments with one another remain within the scope of this disclosure.

Spatial and functional relationships between elements (for example,between modules, circuit elements, semiconductor layers, etc.) aredescribed using various terms, including “connected,” “engaged,”“coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and“disposed.” Unless explicitly described as being “direct,” when arelationship between first and second elements is described in the abovedisclosure, that relationship can be a direct relationship where noother intervening elements are present between the first and secondelements, but can also be an indirect relationship where one or moreintervening elements are present (either spatially or functionally)between the first and second elements. As used herein, the phrase atleast one of A, B, and C should be construed to mean a logical (A OR BOR C), using a non-exclusive logical OR, and should not be construed tomean “at least one of A, at least one of B, and at least one of C.”

In the figures, the direction of an arrow, as indicated by thearrowhead, generally demonstrates the flow of information (such as dataor instructions) that is of interest to the illustration. For example,when element A and element B exchange a variety of information butinformation transmitted from element A to element B is relevant to theillustration, the arrow may point from element A to element B. Thisunidirectional arrow does not imply that no other information istransmitted from element B to element A. Further, for information sentfrom element A to element B, element B may send requests for, or receiptacknowledgements of, the information to element A.

In this application, including the definitions below, the term “module”or the term “controller” may be replaced with the term “circuit.” Theterm “module” may refer to, be part of, or include: an ApplicationSpecific Integrated Circuit (ASIC); a digital, analog, or mixedanalog/digital discrete circuit; a digital, analog, or mixedanalog/digital integrated circuit; a combinational logic circuit; afield programmable gate array (FPGA); a processor circuit (shared,dedicated, or group) that executes code; a memory circuit (shared,dedicated, or group) that stores code executed by the processor circuit;other suitable hardware components that provide the describedfunctionality; or a combination of some or all of the above, such as ina system-on-chip.

The module may include one or more interface circuits. In some examples,the interface circuits may include wired or wireless interfaces that areconnected to a local area network (LAN), the Internet, a wide areanetwork (WAN), or combinations thereof. The functionality of any givenmodule of the present disclosure may be distributed among multiplemodules that are connected via interface circuits. For example, multiplemodules may allow load balancing. In a further example, a server (alsoknown as remote, or cloud) module may accomplish some functionality onbehalf of a client module.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. The term shared processor circuitencompasses a single processor circuit that executes some or all codefrom multiple modules. The term group processor circuit encompasses aprocessor circuit that, in combination with additional processorcircuits, executes some or all code from one or more modules. Referencesto multiple processor circuits encompass multiple processor circuits ondiscrete dies, multiple processor circuits on a single die, multiplecores of a single processor circuit, multiple threads of a singleprocessor circuit, or a combination of the above. The term shared memorycircuit encompasses a single memory circuit that stores some or all codefrom multiple modules. The term group memory circuit encompasses amemory circuit that, in combination with additional memories, storessome or all code from one or more modules.

The term memory circuit is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium may therefore be considered tangible and non-transitory.Non-limiting examples of a non-transitory, tangible computer-readablemedium are nonvolatile memory circuits (such as a flash memory circuit,an erasable programmable read-only memory circuit, or a mask read-onlymemory circuit), volatile memory circuits (such as a static randomaccess memory circuit or a dynamic random access memory circuit),magnetic storage media (such as an analog or digital magnetic tape or ahard disk drive), and optical storage media (such as a CD, a DVD, or aBlu-ray Disc).

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks,flowchart components, and other elements described above serve assoftware specifications, which can be translated into the computerprograms by the routine work of a skilled technician or programmer.

The computer programs include processor-executable instructions that arestored on at least one non-transitory, tangible computer-readablemedium. The computer programs may also include or rely on stored data.The computer programs may encompass a basic input/output system (BIOS)that interacts with hardware of the special purpose computer, devicedrivers that interact with particular devices of the special purposecomputer, one or more operating systems, user applications, backgroundservices, background applications, etc.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language), XML (extensible markuplanguage), or JSON (JavaScript Object Notation) (ii) assembly code,(iii) object code generated from source code by a compiler, (iv) sourcecode for execution by an interpreter, (v) source code for compilationand execution by a just-in-time compiler, etc. As examples only, sourcecode may be written using syntax from languages including C, C++, C#,Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl,Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5threvision), Ada, ASP (Active Server Pages), PHP (PHP: HypertextPreprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, VisualBasic®, Lua, MATLAB, SIMULINK, and Python®.

What is claimed is:
 1. A smart mask comprising: a first material layerconfigured to cover a portion of a face of a person, wherein the portionincludes a mouth of the person; a display connected to the firstmaterial layer and configured to display video images over the mouth; afirst sensor configured to operate as a filter and generate a firstsignal indicative of movement of a first portion of the mouth, whereinthe first sensor includes at least one of a piezoelectric sensor, astrain sensor, a stretch sensor, and a camera; a second sensorconfigured to operate as a filter and generate a second signalindicative of movement of a second portion of the mouth, wherein thesecond sensor includes at least one of a piezoelectric sensor, a strainsensor, a stretch sensor, and a camera; and a control module configuredto control the video images displayed on the display based on the firstsignal and the second signal.
 2. The smart mask of claim 1, furthercomprising: a second material layer; a third material layer; and a powersource connected to the second material layer and powering the controlmodule, wherein the first sensor is connected to the third materiallayer.
 3. The smart mask of claim 2, wherein the first sensor and thesecond sensor comprise adhesive layers configured to removably attachthe first sensor and the second sensor to the face of the person.
 4. Thesmart mask of claim 1, wherein the display is embedded in the firstmaterial layer or overlaid on the first material layer.
 5. The smartmask of claim 1, wherein: the display is disposed in a pocket on thefirst material layer; and the pocket includes an open end through whichthe display is passed when at least one of attaching the display to andremoving the display from the smart mask.
 6. The smart mask of claim 1,further comprising a plurality of displays including the display,wherein each of the plurality of displays is configured to display oneor more video images based on a corresponding movement of the mouth. 7.The smart mask of claim 1, wherein the display is perforated.
 8. Thesmart mask of claim 1, further comprising: a second material layer; anda plurality of spacers disposed to separate the display from the secondmaterial layer.
 9. The smart mask of claim 1, further comprising aplurality of layers including the first material layer, wherein theplurality of layers comprise channels for air flow.
 10. The smart maskof claim 1, further comprising a filter comprising a plurality ofsensors including the first sensor and the second sensor, wherein: theplurality of sensors are configured to generate signals indicative offacial movements of the person including the movement of the firstportion of the mouth and the second portion of the mouth; the signalsindicative of the facial movements include the first signal and thesecond signal; and the control module is configured to display the videoimages based on the signals received from the plurality of sensors. 11.The smart mask of claim 1, wherein the first sensor comprises a graphenesensor.
 12. The smart mask of claim 1, further comprising a plurality ofmulti-purpose sensors, wherein the plurality of multi-purpose sensorseach comprises antimicrobial material.
 13. The smart mask of claim 12,wherein at least one of the plurality of multi-purpose sensors comprisesgraphene-based nanomaterials having antibacterial features.
 14. Thesmart mask of claim 13, wherein the plurality of multi-purpose sensorsinclude at least one of a nanotube sensor and a graphene health sensor.15. The smart mask of claim 1, wherein the display comprises a pluralityof light emitting diodes in a low-resolution grid.
 16. The smart mask ofclaim 1, further comprising an audio sensor configured to detect soundsemanating from the mouth of the person, wherein the control module isconfigured to display the video images based on the detected sounds. 17.The smart mask of claim 1, wherein the control module is configured toperform an artificial training process with one or more devices suchthat at least one of the control module and the one or more devices areable to map detected facial movements of the person to the video imagesfor display on the display.
 18. A system comprising: a first articleimplemented as a facial mask and comprising: a first material layerconfigured to cover a portion of a face of a person, at least one sensorconnected to the first material layer and configured to operate as afilter and generate a sensor signal indicative of at least one movementof a mouth of the person and sound emanating from the mouth of theperson, and a control module configured to transmit the sensor signalfrom the first article via a first transceiver; and a second articleconfigured to overlay the first article and comprising: a secondmaterial layer configured to cover the portion of a body the face of theperson, a second transceiver configured to receive the sensor signal,and at least one display connected to the second material layer andconfigured to display images on the at least one display based on thereceived sensor signal.
 19. The smart mask of claim 1, wherein: thefirst sensor is configured to capture an image of the person; the sensorsignal includes the image; and the control module is configured tocontrol the video images based on the image of the person.
 20. The smartmask of claim 1, wherein the display is perforated and includes aplurality of holes for passage of air.
 21. The smart mask of claim 1,wherein the display is flexible.
 22. The smart mask of claim 1, furthercomprising a transparent pocket, wherein: the display is disposed in thetransparent pocket; and the transparent pocket includes an openingthrough which the display is passed when at least one of attaching thedisplay to and removing the display from the smart mask.
 23. The smartmask of claim 1, further comprising a filter, wherein the first sensorand the second sensor are embedded in the filter.
 24. The smart mask ofclaim 23, wherein the filter comprises, in addition to the first sensorand the second sensor, a porous substrate for passage of airtherethrough.
 25. The smart mask of claim 1, further comprising a stackof porous sheets of material including air channels, wherein each of thestack of porous sheets of material include at least one of graphene, acopper mesh, and a silver mesh.
 26. The smart mask of claim 17, whereinthe control module is configured to, during the artificial trainingprocess, record video images of the face of the person while the personreads preset text.
 27. The smart mask of claim 17, wherein the controlmodule is configured to, during the artificial training process: recordvideo images of the face of the person and corresponding audio while theperson is speaking; convert the recorded audio to text; and map therecorded video images to the text for future facial movementrecognition.
 28. The smart mask of claim 1, wherein: the first sensorincludes at least one of a piezoelectric sensor, a strain sensor, and astretch sensor; and the second sensor includes at least one of apiezoelectric sensor, a strain sensor, and a stretch sensor.
 29. A smartmask comprising: a material layer configured to cover a portion of aface of a person, wherein the portion includes a mouth of the person; aplurality of pockets connected to the material layer; a first displaydisposed in a respective one of the plurality of pockets and configuredto display first video images over a first portion of the mouth; asecond display disposed in a respective one of the plurality of pocketsand configured to display second video images over a second portion ofthe mouth; one or more sensors configured to detect movement of themouth and generate one or more signals indicative of the movement of themouth; and a control module configured to, based on the generated one ormore signals, control the first video images displayed on the firstdisplay and the second video images displayed on the second display. 30.The smart mask of claim 29, wherein the first display and the seconddisplay provide a composite display.
 31. A smart mask comprising: amaterial layer configured to cover a portion of a face of a person,wherein the portion includes a mouth of the person; a display connectedto the material layer and configured to display video images over themouth; a sensor configured to detect a temperature of the face of theperson and generate a signal indicative of the temperature of the face;and a control module configured to control the video images displayed onthe display based on the generated signal to indicate an emotion of theperson based on the detected temperature.
 32. A smart mask comprising: amaterial layer configured to cover a portion of a face of a person,wherein the portion includes a mouth of the person; a display connectedto the material layer and configured to display video images over themouth; a plurality of sensors configured to detect movement of the mouthand generate signals indicative of the movement of the mouth, whereinthe generated signals include sensed values; and a control moduleconfigured to control the video images displayed on the display based onthe generated signals, wherein the control includes at least one of:converting the sensed values to facial landmark positions and displayingthe video images based on the facial landmark positions, performingprincipal component analysis to achieve a target and displaying thevideo images based on the target, and implementing a support vectormachine algorithm where the facial landmark positions represent classesand displaying the video images based on the classes of the faciallandmark positions.